Testing, particularly as applied to computer hardware and software, is an ongoing task in development and maintenance activities which requires a large portion of time, computer and human resources, and effort. Software development may include the development of an entirely new application or program, or the addition of a new feature to an existing application. Software maintenance activities generally include the correction of reported problems. Similarly, for example, hardware development may include the development of a new computer chip, or an improvement to an existing computer chip.
Testing is typically performed to verify proper functioning of new hardware and software, as well as to verify correctness of a modification or an improvement to existing hardware and software.
Typically, test data is used as an input during testing of hardware and software to provide, for example, sample inputs of real-time conditions under which the hardware or software functions to enable problem detection and resolution. During testing, a developer or maintainer typically attempts to debug hardware or software by detecting and solving problems associated with a product prior to the product, for example, reaching a user or customer.
Generally, as the complexity of hardware and software increases, the complexity of testing also increases. For example, as the design of a software module increases in complexity, the number and combination of test cases needed to ensure that each portion of the software module functions correctly may also increase.
Adequately and thoroughly testing hardware and software is typically an important aspect of development and maintenance. In testing, it is important that the test data provide thorough testing coverage, for example, of each component and many or all combinations of components, to adequately test combinations of the components as they may be used outside of the testing environment.
One technique used to generate test data includes manually writing each test data case. For example, each test data case may be manually entered into a file in a computer system using an input device such as a keyboard connected to a terminal.
This technique has several drawbacks. One drawback is that when such data entry is performed manually, it is often cumbersome and time-consuming. Also, the possibility of a data entry error causing incorrect data to be entered and used generally increases with manual data entry. Another drawback is that a human typically constructs each data case. This often results in incomplete testing coverage because a combination may easily be omitted. Particularly as the complexity of the hardware or software being tested increases, the probability that a necessary test case is omitted also increases because the various combinations of elements to be tested increases. In summary, manual data entry is typically costly, inefficient, and prone to human error.
Another method of generating test data includes using a software tool to automatically generate test data. This is an improvement over the previous manual data entry technique because it is more efficient and less prone to human error due to the decreased requirement of human interaction. However, current tools used for this purpose have drawbacks and often lack desirable features and functions.
Generally, tools currently available use random or exhaustive test data generation methods that are associated with an input data set. For example, a random generation method randomly selects data elements in a first input data set to be included in a generated test data case. An exhaustive generation method may, for a single input data set, alternate through each data element. Random test data generation methods can provide broad coverage but may still have significant gaps in coverage. Exhaustive test data generation avoids gaps in coverage, but may not be feasible, or even possible, in circumstances in which the number of possible combinations of input parameters and initial states is intractably large. Thus, there are drawbacks associates with both random and exhaustive test data generation methods.